Rack-level photonic solution

ABSTRACT

A server-to-switch interconnection system that includes a plurality of server modules each having wired server network connection ports, the plurality of server modules being positioned in a server rack. The switch system also includes a rack-level server photonic module that has a plurality of wired network connection ports connecting to a plurality of servers from a single photonic module, an optical transceiver in communication with the wired network connection ports, and an optical port in communication with the optical transceiver.

BACKGROUND

As numbers of computers, particularly servers, are deployed inlarge-scale or hyper-scale data center applications, the need to connectthose computers to one another at massive scale as well as connectingthem to the outside world has driven change in data center networkingtopologies and strategies. Two of the primary drivers of cost andperformance in these large networks are the network topology and thephotonic interconnections between them. The trend has been to utilizemany low-cost low-radix switches connected to other low-radix switchesvia multiple copper and optical connections. As the networks increaseefficiency by increasing data rate, the distances that data signals cantraverse in copper cables diminishes as a result of signal integrityloss in the copper medium. Therefore, the ratio of copper to opticalcables has trended in favor of optical cables, as the signal traversedistance for optical cables is significantly longer.

The fundamental problem with optical cables is cost. Present opticalsolutions, which are cost-effective solutions when used to traverse longdistances, become inefficient when used to traverse shorter distances.As a result, cost-reduction exercises have developed high-channel-countsolutions that amortize the cost of cable attachment and packagingacross a larger number of connections. Where current solutions may useoptical engines with 4 channels or perhaps 8 channels, thesehigh-density solutions favor 24-36 channels.

The remaining problem is the classical last-mile problem, or in thiscase, a last-meter problem. Taking 24-channel or 36-channel cablesdirectly to computer servers is not efficient due to over-provisioning.Likewise, taking 4-channel solutions to many servers is not efficientdue to duplicative packaging costs. As more networks seek to usehigh-radix switches in order to remove layers from the networkhierarchy, they are challenged by the costs of the final layerconnection to the servers. Since the connection between a high-radixmiddle-of-row switch and a large array of servers requires making manyconnections, and the array of servers are typically in differentequipment racks, the problem of requiring the distance capabilities ofoptical connections is conflated with the problem of requiring low-costconnections to many servers.

Therefore, there is a need to minimize wired copper connections toservers to allow for longer data transmission lengths provided by fiberoptical connections, while also minimizing costly optical fiberconnections. Further, legacy servers' output electrical signals and itis desirable to provide a cost-effective system that continues toprovide the ability to use legacy server equipment.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the recited features, advantages and objectsof the present disclosure may be understood in detail, a more particulardescription of the invention, briefly summarized above, may be had byreference to the examples thereof which are illustrated in the appendeddrawings. It is to be noted, however, that the appended drawingsillustrate only typical examples of this invention and are therefore notto be considered limiting of its scope, for the invention may admit toother equally effective examples.

FIG. 1 illustrates an example hyper-scale architecture.

FIG. 2 illustrates an example improved hyper-scale architecture.

FIG. 3 illustrates an example rack-level example architecture schematicusing active optical cable connections.

FIG. 4 illustrates an example rack-level module schematic using acombination of copper and photonics connections.

FIG. 5 illustrates an example vertical rack-level module implementationsuch as would be deployed on rack rails on either side of the servers.

FIG. 6 illustrates an example horizontal rack-level stacked modulephysical implementation.

FIG. 7 illustrates an example connection schematic for a horizontalrack-level stacked module implementation.

FIG. 8 illustrates an example partially-redundant rack-level schematicrequiring fewer optical modules.

FIG. 9 illustrates an example physical connection for apartially-redundant rack-level schematic.

FIG. 10 illustrates an example method for communicating with a pluralityof servers.

DETAILED DESCRIPTION

In the following, reference is made to examples of the inventive conceptof this disclosure. However, it should be understood that the inventiveconcept is not limited to described examples. Instead, any combinationof the following features, elements, or functionalities, whether relatedto different examples or not, is contemplated by the inventors as apossible combination that may be used to implement and practice anaspect of the present innovation. Furthermore, in various examples theinnovation of this disclosure provides numerous advantages over theprior art, and although the examples of the present innovation mayachieve advantages over other possible solutions and/or over the priorart, whether or not a particular advantage is achieved by a givenexample is also not intended to be limiting on the scope of the presentdisclosure. Therefore, the following aspects, features, functionalities,examples, and advantages are intended to be merely illustrative and arenot considered elements or limitations of the appended claims, exceptwhere explicitly recited in a claim. Similarly, reference to “theinvention” or “the innovation” are not to be construed as ageneralization of any inventive subject matter disclosed herein andshall not be considered to be an element or limitation of the appendedclaims except where explicitly recited in a claim.

Example embodiments of the present disclosure provide a rack-levelsolution using CWDM photonic modules to reduce photonic fiberrequirements by a factor of four (four colors per fiber with CWDM).Example embodiments of the disclosure provide mechanical, electrical,power, and cooling solutions in a single rack-level infrastructure thatalso allows hot-swap of the photonic modules.

FIG. 1 illustrates an example hyper-scale architecture 100 having aplurality of server racks 102 each containing a plurality of servers 104therein. Each of the individual servers 104 are electrically connectedto a top of rack (TOR) switch 106, and the TOR switches 106 areoptically connected to middle of row (MOR) switches 110 through photonicconnections 108. The MOR switches 110 are connected (typicallyphotonically) to leaf switches 112 that are connected (typicallyphotonically) to spine switches 114, which in turn may connect to coreswitches. One challenge with hyper scale data centers is that they aretypically housed in in buildings that are kilometers in size, thusresulting in many of the connections in the hyper-scale architecture 100being tens or hundreds of meters long. These distances presentsignificant challenges for signal propagation, as signals degrade asthey traverse the line. Further, as the signal speeds increase, thesignal degradation increases, as faster signals aren't able to traverseas long of a line run as slower signals. For example, at ten gigabitdata transmission speeds the signal can easily propagate through about10 meters copper line connection before degradation starts diminishingthe usable signal quality (depending upon the cable materials andconstruction chosen. At twenty-five gigabit data transmission speeds thesignal can barely propagate three meters of copper successfully, unlessspecial materials are used, and at fifty gigabit the signal transmissionlength in copper is between one meter and two meters, depending on thecable materials and construction, as well as the electronic drivers andreceivers employed on either end of the cable.

FIG. 2 illustrates an example improved hyper-scale architecture 200where the servers 204 are connected directly to the middle of row (MOR)switches 206, which then again connect to the leaf switches 212 and thecore switches 214. In this example architecture the majority of theswitching has been translated into the redundant high radix MOR switchesthat connect directly to the servers 204. However, the physical size ofthe network presents connection issues, as this configuration requiresthe connections to span approximately fifteen meters to connect thevarious components, which is challenging to accomplish with copperconnections and not cost effective to do with photonic connections dueto the sheer number of connections required, which in the example hyperscale architecture 200, would require 1,152 connections (576 primaryconnections and 576 backup/redundant connections to connect the 576racked servers).

FIG. 3 illustrates an example rack-level architecture schematic 300using active optical cable connections to reduce the number of requiredelectrical connections. The servers 302 each have a primary or A port304 and a secondary or B port 306. Each A port 304 is connected to a MORswitch 310 and each B port 306 is connected to a MOR switch 312. Theconnections to the MOR switches 310, 312 are through active opticalcables 308, which may have quad small form-factor pluggable (QFSP) typehot pluggable transceivers on the terminating ends that connect to theservers 302 or the MOR switches 310, 312. This configuration requires anactive optical cable 308 to connect between to each server A port 304and the corresponding MOR switch A 310 and each server B port 306 andthe corresponding MOR switch B 312. Therefore, in the exampleconfiguration shown there would be 48 active optical cables 48 used toconnect to 24 servers 302 each having an A port 304 and a B port 306.

FIG. 4 illustrates an example rack-level module schematic using acombination of copper and photonics connections to facilitatefully-redundant server connectivity. In this example, the failure of asingle MOR switch or the failure of a single module 406 would not lossof connectivity to any servers. The example rack level module schematic400 includes servers 402 again having A and B QFSP ports that areconnected by copper connections 404, to photonics modules 406. Thephotonics modules 406 are then optically connected to the MOR switches408, 412 through optical or photonic cables 410. The optical cables usedin this example can be used to connect to optical modules 406 via CWDMtechniques, reducing the number of required fibers by a factor of 4. Theexample optical modules 406 provide 24 electrical channels per opticalcable, and are thus able to supply 6 servers with 4-channel connections,consistent with Quad Small Form-factor Pluggable (QSFP) connectivity.

Since a typical server rack is one to two feet wide and in the presentexample the photonics modules 406 may be rack-level components, thelongest copper cable connection length will be less than two feet long,thus accommodating high speed gigabit signals, over 50 gigabits, forexample, without significant degradation. The connection from thephotonics modules 406 to the MOR switches 408, 410 may be individualoptical cables having, for example, 24 channels per cable. The photonicsmodule 406 may be a coarse wavelength division multiplexer (CWDM) moduleconfigured to convert the electrical signals received from the servers402 on the copper wires 404 to optical signals that can be transmittedon the photonic cable 410 to the MOR switches 408, 412. The photonicsmodule 406 may provide connections to six QSFP ports, for example,through a single photonic cable 401. The photonic cable 410 may be atwelve wide parallel fiber optical ribbon cable typically used with fourcolors of light to support signal transmission. The twelve fibers allow,as an example, for six send and six receive fibers to be used, and witheach fiber having the ability to carry four distinct optical signalcolors, the twelve wide fiber cable provides a total of 24 channels (6fibers and 4 colors per direction). The twelve-fiber optical cable 410provides 24 electrical channels to be carried across it in opticalsignals, and as such, the photonic module 406 can package send andreceive signals from 6 QFSP ports on the servers 402 for transmissionacross the single 12 wide optical fiber to the MOR switch 408, 412.Although example configurations herein use a twelve wide optical fiberribbon cable, the inventive concepts are not limited to any particularsize, width, or type of optical fiber or connection, as theconfigurations scale up or down easily.

The photonics module 406 may be positioned rack level, i.e., thephotonics module may be built integral to the server rack and thereforepositioned next to, adjacent, or near the servers 402 by being builtdirectly into the server rack that physically supports servers 402, orin a sub-chassis that attaches to the vertical rack rail. This racklevel positioning of the optics module 406 allows for simplification ofthe server QSFP port wiring, as specific wire lengths may be used foreach server port. For example, the wire length for the connection toserver 1 may be shorter than the wire length for the connection toserver 2, thus indicating to a server technician that the wires forserver 1 cannot be plugged into any other server by mistake. This lengthdesignated wire configuration allows for reduced wiring errors andfacilitates efficient and proper connection of server ports to theoptical modules 406. Further, the configuration of the current exampleallows for legacy server technology and configurations to be unchanged,as the send/receive QSFP ports on the servers remain unchanged, thusallowing legacy servers with electrical connections to readily connectto the rack-level optical modules 406 of the present example, thusavoiding the costly process of upgrading to optical or photonic servers.

FIG. 5 illustrates an example vertical rack-level module implementation.The vertical rack-level module 500 may be sized and shaped to be builtintegrally with a server rack that supports a plurality of servers orserver components. More particularly, the module 500 may be sized to beintegrally positioned between the upright support rails or posts of aserver rack on the left and right sides of the server rack, andspecifically, an individual module 500 may be positioned on a side of aserver rack between a front post and a rear post, but generallypositioned near the front rack post with plug, terminals, connectionsproximate corresponding plugs on the server components. The module 500may include a plurality of QSFP or QSFP-DD ports 504 a . . . 504 n, aplurality of optical modules 506 a . . . 506 n, an integrated powersupply 508, and one or more cooling fans 510 all integrally formedtherein. The module 500 may generally extend vertically along a side ofa server rack and have the ports 504 and the optical modules 506positioned to be connected with rows of servers.

In the present example implementation, the first or top set of ports 504a may be configured to connect with the A port of each of the servers502 in the top or adjacent row of the server rack. As discussed above,the wired connections between the QSFP ports 504 and the server A portsmay be of specific lengths to prevent misconnections. For example, awired connection from QFSP port 504 a may be of a specific length thattraverses distance a1, but that is not capable of reaching anotherserver port that is a distance a2 away from the QFSP ports 504 a.Therefore, in order to simplify wiring of the servers and reducemisconnections, the wired connections from QSFP ports 504 a may be threespecific lengths, approximately a1, a2, and a3. This eases connection ofthe servers 502, as the shortest wire of length a1 gets connected to theclosest server 502 A port, the middle length wire of length a2 getsconnected to the middle server 502 A port, and the longest wire oflength a3 gets connected to the left or farthest server 502 A port. TheQFSP ports 504 a are in communication with the optical module 506 awhich converts the electrical signals received from the server 502 intooptical signals that are output from the optical module 506 a andcommunicated to an MOR switch (not shown). The reverse path is followedfor data traveling from an MOR switch to the servers 502, as the opticalsignals are received by the optical module through an optical fiber andare converted to electrical signals within module 506 a that arecommunicated to the QSFP ports 504 and then through wired connections tothe server 502 ports. The server 502 B ports may be connected to amirror module 500 positioned on the left side of the server rack in thesame fashion and set up to connect to the B ports of the server 502.Further, additional rows of servers 502 (not shown) may be connected tothe QSFP ports 504 b . . . 504 n and optical modules 506 b . . . 506 n.

The example vertical rack-level module 500 implementation provides arack-scale solution that sits on or in a rack rail of a server rack.There may be enclosure or unit on each side, left and right, forexample, of a server rack. The enclosure 500 includes an integral powersupply 508 to power active components of the enclosure 500 andexhaust/cooling fans 510. The enclosure 500 includes wired 504 andoptical 506 connections and transceivers to convert signals between theoptical and electrical connections. The example solution provides 24QSFP/QSFP-DD connections capable of, for example, 100G/200G operation.This example configuration requires only eight optical or photoniccables per rack, which is significantly less than the 48 optical cablesrequired in the configuration shown in FIG. 3. Finally, theimplementation of fixed cable lengths reduces cable routing andcomplexity and enhances serviceability.

FIG. 6 illustrates an example horizontal rack-level stacked Enclosurephysical implementation. The server rack 600 includes a plurality ofrows of servers 602 stacked in a rack configuration with horizontalrack-level modules 608 positioned between the server 602 rows to provideelectrical and optical connection thereto. The rack level enclosures 608may be positioned in a server drawer or bay location or may beinterstitially positioned in the rack between two vertically spacedserver drawers, where each drawer contains a row of servers 602. Thehorizontal rack enclosure 608 includes QSFP ports 604 and an optical orphotonic module 606 operation in similar fashion to that described inthe example configuration of FIG. 5. It should be noted that in thisconfiguration, enclosure 608 can be cooled with traditional switch orserver fans housed within the enclosure, and that the units can bepowered by several standard rack-level power solutions including, butnot limited to, pluggable rack power schemes, corded rack power schemes,and AC power schemes. It should also be noted that the optical module606 can be hot-swapped from the enclosure 608 without requiring theremoval or opening of enclosure 608 or the removal of any servers orserver connection cables.

FIG. 7 illustrates an example connection schematic for the horizontalrack-level stacked module implementation shown in FIG. 6. In thisschematic the six QSFP ports 604 on the left side or left half of therack level enclosures 608 are in wired connection to the A ports of eachof the individual servers 602. The QSFP ports 604 communicate theelectrical signals to the photonic module 606 on the left side of therack. The photonic module 606 converts the electric signals to opticalsignals and transmits the optical signals on a multi-fiber communicationmedium 610 to a MOR switch 612. Similarly, the six QSFP ports 604 on theright side or right half of the rack level modules 608 are in wiredconnection to the B ports of each of the individual switches 602. TheQSFP ports 604 communicate the electrical signals to the photonic module606 on the right side of the rack. The photonic module 606 converts theelectric signals to optical signals and transmits the optical signals ona multi-fiber communication medium 610 to a MOR switch 614. The MORswitch 614 may be in optical communication with a plurality of racklevel enclosures 608 through a plurality of individual optical fibercables, of which four are shown in the example configuration of FIG. 7.Therefore, the MOR switch A 612 routes traffic to/from the A ports onthe servers 602 while the MOR switch B 614 routes traffic to/from the Bports on the servers 602. The result of this example configuration isthat the copper connections between the server A or B ports and thecorresponding QSFP ports 604 are very short, typically less than 0.5meters, thus facilitating higher gigabit transmission speeds.

FIG. 8 illustrates an example rack-level schematic 800 showing aplurality of servers 802 having A and B ports, where each A and B portsof the servers 802 are in electrical communication with an optical orphotonic module 806 via wired connectors 804. The optical modules 806are in optical communication with MOR switches 808, 810 by an optical orphotonic fiber connection cable 812. The example schematic 800 providesa 24-channel CWDM photonic module with connections to twelve QSFP portsthrough two photonic cables. Two-lane connections are utilized for eachserver port and both A and B ports are supplied by the same photonicmodule. This configuration requires only eight individual six fiberphotonic cables, where three fibers in the cable are sending and 3fibers are receiving, and each fiber carries four colors for a total oftwelve channels per fiber cable per direction. Therefore, in addition tothe fully-redundant cases using either the horizontal or vertical modulehousing solutions noted above in FIGS. 5-7 that require 8 modules for 24servers, a semi-redundant solution is provided by the example schematicof FIG. 8, as two electrical channels (as opposed to the customary fourelectrical channels) are provided to each server port at the server-endelectrical port connections. As such, only four photonic modules areused to connect 24 servers 602 to the MOR switches 808, 810. In thispartially-redundant case, both the A and B ports of each server areconnected to the same photonic module with two electrical channels foreach connection. As such, the connections from the MOR switches 808, 810are fully redundant so that if one MOR switch 808, 810 fails, allservers will have an alternative path through the second MOR switch 808,810. If it were not redundant, the failure of the A or B switch woulddisrupt connectivity to all 24 servers. The connections from thephotonic module to the servers are not redundant, so if a module fails,connectivity will be lost to six servers, but not to all 24 servers.

The photonic cables 812 are different for the example schematic 800, aseach module must connect to MOR switch A 808 and MOR switch B 810.Therefore, the fiber and channel assignments are arranged accordingly,with three of the six “send” fibers connecting to MOR switch A 808 andthree of the six “send” fibers connecting to MOR switch B 810.Similarly, three of the six “receive” fibers are connected to MOR switchA 808 and three of the six “receive” fibers connecting to MOR switch B810. Since each fiber carries four channels by means of CWDM techniques,twelve channels are provided by three fibers. Note that theseconnections may be constructed with either a custom cable harness or bymeans of a module having two discrete photonic connections, allowing themodule capability to be split between MOR switch A 808 and MOR switch B810. Likewise, at the MOR end, two connectors would allow the samemodule to connect to 12 servers. i.e., 4 shelves of 3 servers each.

FIG. 9 illustrates an example horizontal rack-level stacked modulephysical implementation of the schematic shown in FIG. 8. In the exampleschematic 900, the server 902 A and B ports are connected to rack leveloptical modules 906 that are in communication with MOR switches A and B908, 910 through an optical fiber communication medium 912. In thisexample configuration, MOR switches A and B 908, 910 are opticallyconnected to all four rack level optical modules 906. The optical fibercommunication medium 912 can be, for example, photonic cables with twoseparate cables with two separate connectors at module ends oralternatively a cable harness assembly with two switch ends and a singlemodule end may be used. In this configuration all 24 channels of eachoptical module 906 are used with two channels routed to A ports and 2channels routed to B ports for each of six servers 902.

FIG. 10 illustrates an example method for communicating with a pluralityof servers. The method begins at 1000 and continues to 1002 where datais communicated from a plurality of servers via wired connectors incommunication with a rack level photonic module. The data is generallytransmitted from the servers via network connectors that connect withcopper network cables to communicate the data through the cable. Theother end of the network cable is plugged into an optical or photonicmodule via a QSFP port, for example. At 1004 the optical module convertsthe wired data signals received from the servers into correspondingoptical data signals. The optical data signals are communicated from thephotonic module through an optical fiber to a switch external to theserver rack at 1006. The switch may be a middle of row switch, forexample. The network ports that received the wired signals are incommunication with the optical module and receive signals therefrom.Further, the optical module is positioned at the rack level, i.e.,mounted on the server rack or in the server rack next to the pluralityof servers. The method ends at 1008.

In the preceding, reference is made to examples presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described examples. Instead, any combination of thefollowing features and elements, whether related to different examplesor not, is contemplated to implement and practice contemplated examples.Furthermore, although examples disclosed herein may achieve advantagesover other possible solutions or over the prior art, whether or not aparticular advantage is achieved by a given example is not limiting ofthe scope of the present disclosure. Thus, the preceding aspects,features, examples and advantages are merely illustrative and are notconsidered elements or limitations of the appended claims except whereexplicitly recited in a claim(s).

Examples presented in this disclosure are described above with referenceto flowchart illustrations or block diagrams of methods, apparatus(systems) and computer program products according to examples disclosedherein. It will be understood that each block of the flowchartillustrations or block diagrams, and combinations of blocks in theflowchart illustrations or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general-purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart or block diagram block or blocks.

While the foregoing is directed to examples presented in thisdisclosure, other and further examples or variations may be devisedwithout departing from the basic scope of contemplated examples, and thescope thereof is determined by the claims that follow.

What is claimed is:
 1. A server-to-switch interconnection system,comprising: a plurality of server modules each having wired servernetwork connection ports, the plurality of server modules beingpositioned in a server rack; and a rack-level server photonic modulemounted to a vertical support rail of the server rack or mountedhorizontally in the server rack between rows of the plurality of servermodules, the rack level server photonic module comprising: a pluralityof wired network connection ports connecting the rack-level serverphotonic module to a plurality of servers through wired connections eachhaving a length that is less than a width of the server rack; an opticaltransceiver in communication with the wired network connection ports;and an optical port in optical communication with the opticaltransceiver and a first middle of row switch through a single opticalcable.
 2. The server-to-switch interconnection system of claim 1,further comprising a second middle of row switch in opticalcommunication with the optical port.
 3. The server-to-switchinterconnection system of claim 2, wherein the first middle of rowswitch communicates with A ports on the plurality of server modules andthe second middle of row switch communicates with B ports on theplurality of server modules.
 4. The server-to-switch interconnectionsystem of claim 1, further comprising the server rack having a pluralityof rack level photonic modules, each of the plurality of rack levelphotonic modules being in optical communication with the first middle ofrow switch.
 5. The server-to-switch interconnection system of claim 1,wherein the rack level photonic module further comprises an integratedpower supply.
 6. The server-to-switch interconnection system of claim 1,wherein the rack level photonic module further comprises an integratedcooling fan.
 7. The server-to-switch interconnection system of claim 1,wherein the plurality of wired network connection ports, the opticaltransceiver, and the optical port are contained in a rack level housing.8. The server-to-switch interconnection system of claim 1, wherein theoptical transceiver comprises a 24 channel CWDM photonic module having 6QSFP ports connecting through to the optical port.
 9. Theserver-to-switch interconnection system of claim 1, wherein the singleoptical cable comprises a parallel fiber optical ribbon cable.
 10. Arack level server photonic module, comprising: a plurality of wirednetwork communication ports; and an optical transceiver module connectedto the plurality of wired network communication ports and having anoptical port, the optical transceiver module converting electricalsignals from the plurality of wired network communication ports intooptical signals for transmission to the optical port and convertingoptical signals from the optical port into electrical signals fortransmission to the plurality of wired network communication ports, theplurality or wired network communication ports and the opticaltransceiver module being mounted to a vertical support rail of a serverrack or horizontally in the server rack between rows of servers with theplurality of wired network communication ports being in communicationwith the rows of servers and the optical port being in communicationwith a single optical fiber for communicating with a switch outside theserver rack.
 11. The rack level server photonic module of claim 10,wherein the optical transceiver module is hot swappable.
 12. The racklevel server photonic module of claim 10, wherein the switch outside theserver rack is a middle of row switch.
 13. The rack level serverphotonic module of claim 12, wherein the optical fiber comprises aparallel fiber optical ribbon cable.
 14. The rack level server photonicmodule of claim 13, wherein the plurality of wired network communicationports comprise QSFP ports.
 15. The rack level server photonic module ofclaim 10, wherein wired connections between the optical transceivermodule and the wired network communication ports have a length that isless than a width of the server rack.
 16. A method for communicatingwith a plurality of servers, comprising: communicating data from aplurality of servers via wired connectors in communication with a racklevel photonic module that is integrated into a vertical rail of aserver rack or horizontally between rows of the plurality of servers;converting wired data signals into corresponding optical data signals inthe rack level photonic module; and communicating the optical datasignals from the rack level photonic module through a single opticalcable to a switch external to a server rack containing the plurality ofservers and the rack level photonic module.
 17. The method of claim 16,wherein the wired connectors comprise QSFP ports connected to coppernetwork cables and wherein the optical cable comprises a paralleloptical ribbon cable.